After installation, you must configure the spider pool settings like IP addresses, domains, and pages to be crawled. Some spider pools provide a web interface that allows you to customize crawler frequency, user-agent data, output directory, and indexing filters, which are great features to have when managing the crawl and determining priorities effectively.
Copyright 1995 - . All rights reserved. The content (including but not limited to text, photo, multimedia information, etc) published in this site belongs to China Daily Information Co (CDIC). Without written authorization from CDIC, such content shall not be republished or used in any form. Note: Browsers with 1024*768 or higher resolution are suggested for this site.